Models play an important role in augmented reality (AR) systems to improve the AR experience. Virtual models are used to support algorithms that provide improved occlusion effects, more accurate registration, and improved augmentation of corresponding physical objects. Even with the large body of research exploring modeling techniques for AR systems, the majority of models used by AR systems are created using traditional desktop Computer Aided Design (CAD) systems. AR users are constrained to the role of a consumer of the models with limited capability for creating and modifying them. Two major techniques for virtual objects manipulations have been investigated, namely computer vision and direct manipulation.
Action at a distance is the problem of interacting with virtual objects that are out of arm's reach. One example is the AR working planes technique by Piekarski and Thomas (W. Piekarski and B. H. Thomas, “Augmented reality working planes: a foundation for action and construction at a distance,” ISMAR 2004, pp. 162-171, 2004) to create polygonal models of outdoor buildings using wearable computers. The technique defines an outline model of a building by intersecting a collection of planes, which are created by the user sighting along the surfaces of the building to be modeled. Bastian et al. (J. Bastian, B. Ward, R. Hill, A. van den Hengel, and A. Dick, “Interactive modeling for AR applications,” in ISMAR 2010, pp. 199-205) developed a modeling approach using segmentation algorithm from computer vision. The 3D shape of the object is reconstructed based on the silhouette of the object from various angles, which has been segmented from the background. Computer vision techniques cannot model purely virtual objects, i.e. there is a requirement of a physical object to model against. AR modeling techniques, however, are still facing the challenge of precision, which is an established advantage of traditional CAD programs.
A major challenge for AR systems, especially for use outdoors, is the development of intuitive input devices. The traditional keyboard and mouse are not suitable due to the mobile nature of the outdoor AR system. Immense research effort in the area of input devices has produced a range of devices in various form factors, with different types of sensors used for a diverse set of techniques. One form of input device that has been investigated is glove-based input devices for their support of natural, intuitive, and hands-free interaction. There are two main forms of sensing for gloves. Firstly there are pinch gloves that sense when the tips of the fingers come in contact with the thumb, i.e. a pinch gesture. The second form is a data glove that senses the joint angles of one or more of the user's fingers.
Historically, data gloves have been used in AR/VR systems for direct manipulation tasks. For example, when a user reaches out and grasps a virtual cup, the system senses the shape of the user's hand to determine the user's action. In one system the joints of the user's index finger were tracked via a computer vision system which sensed strategically placed ring- and ball-shaped retroreflective markers on the user's hand. The positional data of the joints was combined with the kinetic models of the joints to allow gesture tracking of the finger to perform picking gesture on virtual objects. Another vision-based approach used hand tracking of a pair of custom-patterned gloves. The camera interpolates the current hand pose from the unique pattern detected from the captured frame.
Pinch glove input devices incorporate conductive fabric pads on the tips of each finger and thumbs into a pair of gloves. One example of a pinch glove sensor is the chording glove. This glove is a text input device for wearable systems that mounts sensors on the fingers and various positions on the hand. The user presses on a combination of sensors to generate characters based on a chording keymap. Another system that used pinch gloves is the Tinmith wearable AR system (W. Piekarski and B. H. Thomas, “Interactive Augmented Reality Techniques for Construction at a Distance of 3D Geometry,” 7th Int'l Workshop on Immersive Projection Technology/9th Eurographics Workshop on Virtual Environments, 2003). The Tinmith system uses a pair of pinch gloves with fiducial markers on the thumbs for cursor-based manipulation techniques. Pinching input events are mapped to a set of menu items to control the system, placed on the left and right bottom corners of the screen. The pinch gloves are connected to the AR system via Bluetooth. The head mounted camera detects the markers to control two cursors on the user's viewport.
However most of the glove based input devices require either position tracking of the hand which can be computationally challenging, or require unnatural and fatiguing movements. For example in the Tinmith system described above, users experienced fatigue and discomfort associated with holding the thumb markers up at an unnatural pose for extended periods of time. Secondly the range of the cursor movements was also limited by the available screen space, which affected the precision of the manipulation.
There is thus a need to provide improved user input devices and input methods for wearable augmented reality systems, or to at least provide a useful alternative to existing systems.